StreamCorder: Fast Trial-and-Error Analysis in Scientific Databases

Scientific databases managing continuous observations of natural phenomena are currently storing multiple TBs of data and will soon reach the PBs range. Given their size, a very important part of the system load comes from scientists sifting through long stretches of noise in search of relevant events. This is a trial-and-error procedure where a scientist retrieves a data interval, processes it, determines whether the result of the analysis is acceptable, repeats the analysis if that is not the case or proceeds with the next data interval. In general, the large volume of necessary data-shipping and -processing inhibits interactive analysis. As a result, scientists are forced to coordinate lengthy offline event searches with online evaluation sessions.